{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute this cell to install dependencies\n",
    "%pip install sf-hamilton[visualization]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kedro to Hamilton plugin [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dagworks-inc/hamilton/blob/main/examples/kedro/kedro-plugin/kedro_to_hamilton.ipynb) [![GitHub badge](https://img.shields.io/badge/github-view_source-2b3137?logo=github)](https://github.com/apache/hamilton/blob/main/examples/kedro/kedro-plugin/kedro_to_hamilton.ipynb)\n",
    "\n",
    "\n",
    "The plugin allows you to convert a Kedro `Pipeline` object in to a valid Hamilton `Driver`. This allows you to execute your Kedro pipeline on Hamilton and track execution using the [Hamilton UI](https://hamilton.apache.org/concepts/ui/), which provides rich observability and introspection features.\n",
    "\n",
    "\n",
    "## Content\n",
    "1. From Kedro `Pipeline` to Hamilton `Driver`\n",
    "2. Executing the `Driver`\n",
    "3. Reusing your Kedro `DataCatalog`\n",
    "4. Using Hamilton materialization\n",
    "5. Connect to the Hamilton UI\n",
    "\n",
    "## Prerequisite\n",
    "Install the example Kedro project by executing the next cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install ../kedro-code\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. From `Pipeline` to `Driver`\n",
    "We import from the Kedro project `kedro_code` the modules `data_processing` to create a `Pipeline` object. Then, we pass it to the Hamilton plugin using `h_kedro.kedro_pipeline_to_driver()` to create a Hamilton `Driver`. It can be viewed, executed, and supports all `Driver` operations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"639pt\" height=\"358pt\"\n",
       " viewBox=\"0.00 0.00 639.00 358.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 354)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-354 635,-354 635,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"46,-210 46,-342 142,-342 142,-210 46,-210\"/>\n",
       "<text text-anchor=\"middle\" x=\"94\" y=\"-326.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- preprocessed_shuttles -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>preprocessed_shuttles</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M415,-64C415,-64 239,-64 239,-64 233,-64 227,-58 227,-52 227,-52 227,-12 227,-12 227,-6 233,0 239,0 239,0 415,0 415,0 421,0 427,-6 427,-12 427,-12 427,-52 427,-52 427,-58 421,-64 415,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"238\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">preprocessed_shuttles</text>\n",
       "<text text-anchor=\"start\" x=\"288.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- model_input_table -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>model_input_table</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M619,-137C619,-137 478,-137 478,-137 472,-137 466,-131 466,-125 466,-125 466,-85 466,-85 466,-79 472,-73 478,-73 478,-73 619,-73 619,-73 625,-73 631,-79 631,-85 631,-85 631,-125 631,-125 631,-131 625,-137 619,-137\"/>\n",
       "<text text-anchor=\"start\" x=\"477\" y=\"-115.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">model_input_table</text>\n",
       "<text text-anchor=\"start\" x=\"510\" y=\"-87.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- preprocessed_shuttles&#45;&gt;model_input_table -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>preprocessed_shuttles&#45;&gt;model_input_table</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M424.28,-64.02C434.86,-67.54 445.61,-71.11 456.13,-74.61\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"455.12,-77.97 465.72,-77.8 457.33,-71.32 455.12,-77.97\"/>\n",
       "</g>\n",
       "<!-- preprocessed_companies -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>preprocessed_companies</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M425,-210C425,-210 229,-210 229,-210 223,-210 217,-204 217,-198 217,-198 217,-158 217,-158 217,-152 223,-146 229,-146 229,-146 425,-146 425,-146 431,-146 437,-152 437,-158 437,-158 437,-198 437,-198 437,-204 431,-210 425,-210\"/>\n",
       "<text text-anchor=\"start\" x=\"228\" y=\"-188.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">preprocessed_companies</text>\n",
       "<text text-anchor=\"start\" x=\"288.5\" y=\"-160.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- preprocessed_companies&#45;&gt;model_input_table -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>preprocessed_companies&#45;&gt;model_input_table</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M424.28,-145.98C434.86,-142.46 445.61,-138.89 456.13,-135.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"457.33,-138.68 465.72,-132.2 455.12,-132.03 457.33,-138.68\"/>\n",
       "</g>\n",
       "<!-- _preprocessed_shuttles_inputs -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>_preprocessed_shuttles_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"178,-54.5 10,-54.5 10,-9.5 178,-9.5 178,-54.5\"/>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-27.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">shuttles</text>\n",
       "<text text-anchor=\"start\" x=\"87\" y=\"-27.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- _preprocessed_shuttles_inputs&#45;&gt;preprocessed_shuttles -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_preprocessed_shuttles_inputs&#45;&gt;preprocessed_shuttles</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M178.07,-32C190.52,-32 203.52,-32 216.42,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"216.73,-35.5 226.73,-32 216.73,-28.5 216.73,-35.5\"/>\n",
       "</g>\n",
       "<!-- _model_input_table_inputs -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>_model_input_table_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"409.5,-127.5 244.5,-127.5 244.5,-82.5 409.5,-82.5 409.5,-127.5\"/>\n",
       "<text text-anchor=\"start\" x=\"260\" y=\"-100.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">reviews</text>\n",
       "<text text-anchor=\"start\" x=\"319\" y=\"-100.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- _model_input_table_inputs&#45;&gt;model_input_table -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>_model_input_table_inputs&#45;&gt;model_input_table</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M409.73,-105C424.59,-105 440.18,-105 455.31,-105\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"455.7,-108.5 465.7,-105 455.7,-101.5 455.7,-108.5\"/>\n",
       "</g>\n",
       "<!-- _preprocessed_companies_inputs -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>_preprocessed_companies_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"188,-200.5 0,-200.5 0,-155.5 188,-155.5 188,-200.5\"/>\n",
       "<text text-anchor=\"start\" x=\"15\" y=\"-173.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">companies</text>\n",
       "<text text-anchor=\"start\" x=\"97\" y=\"-173.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- _preprocessed_companies_inputs&#45;&gt;preprocessed_companies -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>_preprocessed_companies_inputs&#45;&gt;preprocessed_companies</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M188.29,-178C194.31,-178 200.41,-178 206.52,-178\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"206.81,-181.5 216.81,-178 206.81,-174.5 206.81,-181.5\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"123.5,-310.5 64.5,-310.5 64.5,-273.5 123.5,-273.5 123.5,-310.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"94\" y=\"-288.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M122,-255.5C122,-255.5 66,-255.5 66,-255.5 60,-255.5 54,-249.5 54,-243.5 54,-243.5 54,-230.5 54,-230.5 54,-224.5 60,-218.5 66,-218.5 66,-218.5 122,-218.5 122,-218.5 128,-218.5 134,-224.5 134,-230.5 134,-230.5 134,-243.5 134,-243.5 134,-249.5 128,-255.5 122,-255.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"94\" y=\"-233.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<hamilton.driver.Driver at 0x7f269613acd0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from hamilton.plugins import h_kedro\n",
    "from kedro_code.pipelines import data_processing\n",
    "\n",
    "dr = h_kedro.kedro_pipeline_to_driver(data_processing.create_pipeline())\n",
    "dr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also pass multiple Kedro `Pipeline` to compose them together in a single Hamilton `Driver`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"1220pt\" height=\"372pt\"\n",
       " viewBox=\"0.00 0.00 1220.00 372.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 368)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-368 1216,-368 1216,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"46,-224 46,-356 142,-356 142,-224 46,-224\"/>\n",
       "<text text-anchor=\"middle\" x=\"94\" y=\"-340.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- preprocessed_shuttles -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>preprocessed_shuttles</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M415,-78C415,-78 239,-78 239,-78 233,-78 227,-72 227,-66 227,-66 227,-26 227,-26 227,-20 233,-14 239,-14 239,-14 415,-14 415,-14 421,-14 427,-20 427,-26 427,-26 427,-66 427,-66 427,-72 421,-78 415,-78\"/>\n",
       "<text text-anchor=\"start\" x=\"238\" y=\"-56.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">preprocessed_shuttles</text>\n",
       "<text text-anchor=\"start\" x=\"288.5\" y=\"-28.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- model_input_table -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>model_input_table</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M619,-151C619,-151 478,-151 478,-151 472,-151 466,-145 466,-139 466,-139 466,-99 466,-99 466,-93 472,-87 478,-87 478,-87 619,-87 619,-87 625,-87 631,-93 631,-99 631,-99 631,-139 631,-139 631,-145 625,-151 619,-151\"/>\n",
       "<text text-anchor=\"start\" x=\"477\" y=\"-129.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">model_input_table</text>\n",
       "<text text-anchor=\"start\" x=\"510\" y=\"-101.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- preprocessed_shuttles&#45;&gt;model_input_table -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>preprocessed_shuttles&#45;&gt;model_input_table</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M424.28,-78.02C434.86,-81.54 445.61,-85.11 456.13,-88.61\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"455.12,-91.97 465.72,-91.8 457.33,-85.32 455.12,-91.97\"/>\n",
       "</g>\n",
       "<!-- X_test -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>X_test</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M850,-310C850,-310 803,-310 803,-310 797,-310 791,-304 791,-298 791,-298 791,-258 791,-258 791,-252 797,-246 803,-246 803,-246 850,-246 850,-246 856,-246 862,-252 862,-258 862,-258 862,-298 862,-298 862,-304 856,-310 850,-310\"/>\n",
       "<text text-anchor=\"start\" x=\"802\" y=\"-288.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">X_test</text>\n",
       "<text text-anchor=\"start\" x=\"812.5\" y=\"-260.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Any</text>\n",
       "</g>\n",
       "<!-- evaluate_model -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>evaluate_model</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M1200,-172C1200,-172 1078,-172 1078,-172 1072,-172 1066,-166 1066,-160 1066,-160 1066,-120 1066,-120 1066,-114 1072,-108 1078,-108 1078,-108 1200,-108 1200,-108 1206,-108 1212,-114 1212,-120 1212,-120 1212,-160 1212,-160 1212,-166 1206,-172 1200,-172\"/>\n",
       "<text text-anchor=\"start\" x=\"1077\" y=\"-150.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">evaluate_model</text>\n",
       "<text text-anchor=\"start\" x=\"1104\" y=\"-122.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">NoneType</text>\n",
       "</g>\n",
       "<!-- X_test&#45;&gt;evaluate_model -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>X_test&#45;&gt;evaluate_model</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M862.21,-262.57C908.97,-241.79 993.61,-204.17 1056.83,-176.07\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1058.28,-179.26 1066,-172 1055.44,-172.86 1058.28,-179.26\"/>\n",
       "</g>\n",
       "<!-- X_train -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>X_train</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M853,-228C853,-228 800,-228 800,-228 794,-228 788,-222 788,-216 788,-216 788,-176 788,-176 788,-170 794,-164 800,-164 800,-164 853,-164 853,-164 859,-164 865,-170 865,-176 865,-176 865,-216 865,-216 865,-222 859,-228 853,-228\"/>\n",
       "<text text-anchor=\"start\" x=\"799\" y=\"-206.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">X_train</text>\n",
       "<text text-anchor=\"start\" x=\"812.5\" y=\"-178.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Any</text>\n",
       "</g>\n",
       "<!-- regressor -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>regressor</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M1025,-172C1025,-172 906,-172 906,-172 900,-172 894,-166 894,-160 894,-160 894,-120 894,-120 894,-114 900,-108 906,-108 906,-108 1025,-108 1025,-108 1031,-108 1037,-114 1037,-120 1037,-120 1037,-160 1037,-160 1037,-166 1031,-172 1025,-172\"/>\n",
       "<text text-anchor=\"start\" x=\"927.5\" y=\"-150.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">regressor</text>\n",
       "<text text-anchor=\"start\" x=\"905\" y=\"-122.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">LinearRegression</text>\n",
       "</g>\n",
       "<!-- X_train&#45;&gt;regressor -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>X_train&#45;&gt;regressor</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M865.3,-180.55C871.31,-178.09 877.72,-175.47 884.28,-172.79\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"886.04,-175.85 893.97,-168.83 883.39,-169.37 886.04,-175.85\"/>\n",
       "</g>\n",
       "<!-- regressor&#45;&gt;evaluate_model -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>regressor&#45;&gt;evaluate_model</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1037.25,-140C1043.27,-140 1049.41,-140 1055.52,-140\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1055.75,-143.5 1065.75,-140 1055.75,-136.5 1055.75,-143.5\"/>\n",
       "</g>\n",
       "<!-- y_test -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>y_test</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M849.5,-64C849.5,-64 803.5,-64 803.5,-64 797.5,-64 791.5,-58 791.5,-52 791.5,-52 791.5,-12 791.5,-12 791.5,-6 797.5,0 803.5,0 803.5,0 849.5,0 849.5,0 855.5,0 861.5,-6 861.5,-12 861.5,-12 861.5,-52 861.5,-52 861.5,-58 855.5,-64 849.5,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"802.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">y_test</text>\n",
       "<text text-anchor=\"start\" x=\"812.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Any</text>\n",
       "</g>\n",
       "<!-- y_test&#45;&gt;evaluate_model -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>y_test&#45;&gt;evaluate_model</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M861.51,-42.01C903.18,-54.47 975.79,-76.82 1037,-99 1043.34,-101.3 1049.86,-103.75 1056.41,-106.26\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1055.4,-109.62 1065.98,-109.99 1057.93,-103.1 1055.4,-109.62\"/>\n",
       "</g>\n",
       "<!-- split_data -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>split_data</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M747,-187C747,-187 672,-187 672,-187 666,-187 660,-181 660,-175 660,-175 660,-135 660,-135 660,-129 666,-123 672,-123 672,-123 747,-123 747,-123 753,-123 759,-129 759,-135 759,-135 759,-175 759,-175 759,-181 753,-187 747,-187\"/>\n",
       "<text text-anchor=\"start\" x=\"671\" y=\"-165.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">split_data</text>\n",
       "<text text-anchor=\"start\" x=\"695.5\" y=\"-137.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Dict</text>\n",
       "</g>\n",
       "<!-- model_input_table&#45;&gt;split_data -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>model_input_table&#45;&gt;split_data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M631.33,-137.53C637.57,-138.94 643.79,-140.35 649.84,-141.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"649.3,-145.18 659.82,-143.98 650.84,-138.36 649.3,-145.18\"/>\n",
       "</g>\n",
       "<!-- preprocessed_companies -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>preprocessed_companies</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M425,-224C425,-224 229,-224 229,-224 223,-224 217,-218 217,-212 217,-212 217,-172 217,-172 217,-166 223,-160 229,-160 229,-160 425,-160 425,-160 431,-160 437,-166 437,-172 437,-172 437,-212 437,-212 437,-218 431,-224 425,-224\"/>\n",
       "<text text-anchor=\"start\" x=\"228\" y=\"-202.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">preprocessed_companies</text>\n",
       "<text text-anchor=\"start\" x=\"288.5\" y=\"-174.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- preprocessed_companies&#45;&gt;model_input_table -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>preprocessed_companies&#45;&gt;model_input_table</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M424.28,-159.98C434.86,-156.46 445.61,-152.89 456.13,-149.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"457.33,-152.68 465.72,-146.2 455.12,-146.03 457.33,-152.68\"/>\n",
       "</g>\n",
       "<!-- y_train -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>y_train</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M852.5,-146C852.5,-146 800.5,-146 800.5,-146 794.5,-146 788.5,-140 788.5,-134 788.5,-134 788.5,-94 788.5,-94 788.5,-88 794.5,-82 800.5,-82 800.5,-82 852.5,-82 852.5,-82 858.5,-82 864.5,-88 864.5,-94 864.5,-94 864.5,-134 864.5,-134 864.5,-140 858.5,-146 852.5,-146\"/>\n",
       "<text text-anchor=\"start\" x=\"799.5\" y=\"-124.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">y_train</text>\n",
       "<text text-anchor=\"start\" x=\"812.5\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Any</text>\n",
       "</g>\n",
       "<!-- y_train&#45;&gt;regressor -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>y_train&#45;&gt;regressor</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M864.57,-121.04C870.7,-122.2 877.25,-123.44 883.96,-124.71\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"883.39,-128.17 893.87,-126.6 884.7,-121.29 883.39,-128.17\"/>\n",
       "</g>\n",
       "<!-- split_data&#45;&gt;X_test -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>split_data&#45;&gt;X_test</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M740.71,-187.32C755.37,-202.99 773.11,-221.97 788.57,-238.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"786.1,-240.98 795.49,-245.9 791.21,-236.2 786.1,-240.98\"/>\n",
       "</g>\n",
       "<!-- split_data&#45;&gt;X_train -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>split_data&#45;&gt;X_train</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M759.27,-172.39C765.54,-174.62 771.95,-176.91 778.19,-179.13\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"777.23,-182.5 787.82,-182.57 779.58,-175.91 777.23,-182.5\"/>\n",
       "</g>\n",
       "<!-- split_data&#45;&gt;y_test -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>split_data&#45;&gt;y_test</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M740.71,-122.68C755.37,-107.01 773.11,-88.03 788.57,-71.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"791.21,-73.8 795.49,-64.1 786.1,-69.02 791.21,-73.8\"/>\n",
       "</g>\n",
       "<!-- split_data&#45;&gt;y_train -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>split_data&#45;&gt;y_train</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M759.27,-137.61C765.62,-135.35 772.11,-133.03 778.43,-130.78\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"779.93,-133.96 788.17,-127.31 777.58,-127.37 779.93,-133.96\"/>\n",
       "</g>\n",
       "<!-- _preprocessed_shuttles_inputs -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>_preprocessed_shuttles_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"178,-68.5 10,-68.5 10,-23.5 178,-23.5 178,-68.5\"/>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-41.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">shuttles</text>\n",
       "<text text-anchor=\"start\" x=\"87\" y=\"-41.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- _preprocessed_shuttles_inputs&#45;&gt;preprocessed_shuttles -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_preprocessed_shuttles_inputs&#45;&gt;preprocessed_shuttles</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M178.07,-46C190.52,-46 203.52,-46 216.42,-46\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"216.73,-49.5 226.73,-46 216.73,-42.5 216.73,-49.5\"/>\n",
       "</g>\n",
       "<!-- _model_input_table_inputs -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>_model_input_table_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"409.5,-141.5 244.5,-141.5 244.5,-96.5 409.5,-96.5 409.5,-141.5\"/>\n",
       "<text text-anchor=\"start\" x=\"260\" y=\"-114.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">reviews</text>\n",
       "<text text-anchor=\"start\" x=\"319\" y=\"-114.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- _model_input_table_inputs&#45;&gt;model_input_table -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>_model_input_table_inputs&#45;&gt;model_input_table</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M409.73,-119C424.59,-119 440.18,-119 455.31,-119\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"455.7,-122.5 465.7,-119 455.7,-115.5 455.7,-122.5\"/>\n",
       "</g>\n",
       "<!-- _preprocessed_companies_inputs -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>_preprocessed_companies_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"188,-214.5 0,-214.5 0,-169.5 188,-169.5 188,-214.5\"/>\n",
       "<text text-anchor=\"start\" x=\"15\" y=\"-187.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">companies</text>\n",
       "<text text-anchor=\"start\" x=\"97\" y=\"-187.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- _preprocessed_companies_inputs&#45;&gt;preprocessed_companies -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>_preprocessed_companies_inputs&#45;&gt;preprocessed_companies</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M188.29,-192C194.31,-192 200.41,-192 206.52,-192\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"206.81,-195.5 216.81,-192 206.81,-188.5 206.81,-195.5\"/>\n",
       "</g>\n",
       "<!-- _split_data_inputs -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>_split_data_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"631,-214.5 466,-214.5 466,-169.5 631,-169.5 631,-214.5\"/>\n",
       "<text text-anchor=\"start\" x=\"481.5\" y=\"-187.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">model_options</text>\n",
       "<text text-anchor=\"start\" x=\"588.5\" y=\"-187.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Dict</text>\n",
       "</g>\n",
       "<!-- _split_data_inputs&#45;&gt;split_data -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>_split_data_inputs&#45;&gt;split_data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M631.33,-172.96C637.57,-171.51 643.79,-170.06 649.84,-168.65\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"650.88,-172 659.82,-166.33 649.29,-165.19 650.88,-172\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"123.5,-324.5 64.5,-324.5 64.5,-287.5 123.5,-287.5 123.5,-324.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"94\" y=\"-302.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M122,-269.5C122,-269.5 66,-269.5 66,-269.5 60,-269.5 54,-263.5 54,-257.5 54,-257.5 54,-244.5 54,-244.5 54,-238.5 60,-232.5 66,-232.5 66,-232.5 122,-232.5 122,-232.5 128,-232.5 134,-238.5 134,-244.5 134,-244.5 134,-257.5 134,-257.5 134,-263.5 128,-269.5 122,-269.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"94\" y=\"-247.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<hamilton.driver.Driver at 0x7f267bb30610>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from hamilton.plugins import h_kedro\n",
    "from kedro_code.pipelines import data_processing, data_science\n",
    "\n",
    "dr = h_kedro.kedro_pipeline_to_driver(\n",
    "    data_processing.create_pipeline(),\n",
    "    data_science.create_pipeline(),\n",
    ")\n",
    "dr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Execute the `Driver`\n",
    "You can execute `Driver` using the regular Hamilton approach. We load data from files and pass it to `Driver.execute(inputs=...)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shuttle_location</th>\n",
       "      <th>shuttle_type</th>\n",
       "      <th>engine_type</th>\n",
       "      <th>engine_vendor</th>\n",
       "      <th>engines</th>\n",
       "      <th>passenger_capacity</th>\n",
       "      <th>cancellation_policy</th>\n",
       "      <th>crew</th>\n",
       "      <th>d_check_complete</th>\n",
       "      <th>moon_clearance_complete</th>\n",
       "      <th>...</th>\n",
       "      <th>review_scores_crew</th>\n",
       "      <th>review_scores_location</th>\n",
       "      <th>review_scores_price</th>\n",
       "      <th>number_of_reviews</th>\n",
       "      <th>reviews_per_month</th>\n",
       "      <th>id</th>\n",
       "      <th>company_rating</th>\n",
       "      <th>company_location</th>\n",
       "      <th>total_fleet_count</th>\n",
       "      <th>iata_approved</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sao Tome and Principe</td>\n",
       "      <td>Type V5</td>\n",
       "      <td>Plasma</td>\n",
       "      <td>ThetaBase Services</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4</td>\n",
       "      <td>moderate</td>\n",
       "      <td>2.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>26</td>\n",
       "      <td>0.77</td>\n",
       "      <td>32413</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Faroe Islands</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Wallis and Futuna</td>\n",
       "      <td>Type V2</td>\n",
       "      <td>Plasma</td>\n",
       "      <td>ThetaBase Services</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "      <td>moderate</td>\n",
       "      <td>3.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>61</td>\n",
       "      <td>0.62</td>\n",
       "      <td>14122</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Malta</td>\n",
       "      <td>1.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Niue</td>\n",
       "      <td>Type F5</td>\n",
       "      <td>Quantum</td>\n",
       "      <td>ThetaBase Services</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>strict</td>\n",
       "      <td>1.0</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>467</td>\n",
       "      <td>4.66</td>\n",
       "      <td>47761</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Niue</td>\n",
       "      <td>2.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Malta</td>\n",
       "      <td>Type V2</td>\n",
       "      <td>Quantum</td>\n",
       "      <td>ThetaBase Services</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>moderate</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>318</td>\n",
       "      <td>3.22</td>\n",
       "      <td>26648</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Niue</td>\n",
       "      <td>2.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Malta</td>\n",
       "      <td>Type V2</td>\n",
       "      <td>Plasma</td>\n",
       "      <td>ThetaBase Services</td>\n",
       "      <td>5.0</td>\n",
       "      <td>10</td>\n",
       "      <td>strict</td>\n",
       "      <td>5.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>22</td>\n",
       "      <td>0.29</td>\n",
       "      <td>26648</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Niue</td>\n",
       "      <td>2.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        shuttle_location shuttle_type engine_type       engine_vendor  \\\n",
       "0  Sao Tome and Principe      Type V5      Plasma  ThetaBase Services   \n",
       "1      Wallis and Futuna      Type V2      Plasma  ThetaBase Services   \n",
       "2                   Niue      Type F5     Quantum  ThetaBase Services   \n",
       "3                  Malta      Type V2     Quantum  ThetaBase Services   \n",
       "4                  Malta      Type V2      Plasma  ThetaBase Services   \n",
       "\n",
       "   engines  passenger_capacity cancellation_policy  crew  d_check_complete  \\\n",
       "0      2.0                   4            moderate   2.0             False   \n",
       "1      3.0                   5            moderate   3.0             False   \n",
       "2      1.0                   2              strict   1.0              True   \n",
       "3      1.0                   2            moderate   1.0             False   \n",
       "4      5.0                  10              strict   5.0             False   \n",
       "\n",
       "   moon_clearance_complete  ...  review_scores_crew  review_scores_location  \\\n",
       "0                    False  ...                 9.0                     9.0   \n",
       "1                    False  ...                10.0                    10.0   \n",
       "2                    False  ...                10.0                    10.0   \n",
       "3                    False  ...                10.0                     9.0   \n",
       "4                    False  ...                10.0                     9.0   \n",
       "\n",
       "   review_scores_price  number_of_reviews  reviews_per_month     id  \\\n",
       "0                  9.0                 26               0.77  32413   \n",
       "1                  9.0                 61               0.62  14122   \n",
       "2                 10.0                467               4.66  47761   \n",
       "3                  9.0                318               3.22  26648   \n",
       "4                 10.0                 22               0.29  26648   \n",
       "\n",
       "   company_rating  company_location  total_fleet_count  iata_approved  \n",
       "0             1.0     Faroe Islands                1.0          False  \n",
       "1             1.0             Malta                1.0           True  \n",
       "2             1.0              Niue                2.0          False  \n",
       "3             1.0              Niue                2.0           True  \n",
       "4             1.0              Niue                2.0           True  \n",
       "\n",
       "[5 rows x 27 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# loading stored data\n",
    "inputs=dict(\n",
    "    companies=pd.read_csv(\"../kedro-code/data/01_raw/companies.csv\"),\n",
    "    reviews=pd.read_csv(\"../kedro-code/data/01_raw/reviews.csv\"),\n",
    "    shuttles=pd.read_excel(\"../kedro-code/data/01_raw/shuttles.xlsx\"),\n",
    ")\n",
    "results = dr.execute([\"model_input_table\"], inputs=inputs)\n",
    "results[\"model_input_table\"].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Reuse the Kedro `DataCatalog`\n",
    "If you have an existing Kedro project with a `DataCatalog` (probably in a `.yaml` file), you can directly reuse it. In Hamilton, both datasets and \"parameters\" can be passed to `Driver.execute(inputs=...)`. \n",
    "\n",
    "To do so, you need to load the Kedro project and access the catalog and parameters from the `KedroSession`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[05/22/24 16:10:54] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">companies</span> <span style=\"font-weight: bold\">(</span>CSVDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                        <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m[05/22/24 16:10:54]\u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mcompanies\u001B[0m \u001B[1m(\u001B[0mCSVDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                        \u001B]8;id=554145;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=747028;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">reviews</span> <span style=\"font-weight: bold\">(</span>CSVDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                          <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m                   \u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mreviews\u001B[0m \u001B[1m(\u001B[0mCSVDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                          \u001B]8;id=756925;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=7965;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">shuttles</span> <span style=\"font-weight: bold\">(</span>ExcelDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                       <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m                   \u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mshuttles\u001B[0m \u001B[1m(\u001B[0mExcelDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                       \u001B]8;id=791772;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=315027;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shuttle_location</th>\n",
       "      <th>shuttle_type</th>\n",
       "      <th>engine_type</th>\n",
       "      <th>engine_vendor</th>\n",
       "      <th>engines</th>\n",
       "      <th>passenger_capacity</th>\n",
       "      <th>cancellation_policy</th>\n",
       "      <th>crew</th>\n",
       "      <th>d_check_complete</th>\n",
       "      <th>moon_clearance_complete</th>\n",
       "      <th>...</th>\n",
       "      <th>review_scores_crew</th>\n",
       "      <th>review_scores_location</th>\n",
       "      <th>review_scores_price</th>\n",
       "      <th>number_of_reviews</th>\n",
       "      <th>reviews_per_month</th>\n",
       "      <th>id</th>\n",
       "      <th>company_rating</th>\n",
       "      <th>company_location</th>\n",
       "      <th>total_fleet_count</th>\n",
       "      <th>iata_approved</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sao Tome and Principe</td>\n",
       "      <td>Type V5</td>\n",
       "      <td>Plasma</td>\n",
       "      <td>ThetaBase Services</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4</td>\n",
       "      <td>moderate</td>\n",
       "      <td>2.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>26</td>\n",
       "      <td>0.77</td>\n",
       "      <td>32413</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Faroe Islands</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Wallis and Futuna</td>\n",
       "      <td>Type V2</td>\n",
       "      <td>Plasma</td>\n",
       "      <td>ThetaBase Services</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "      <td>moderate</td>\n",
       "      <td>3.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>61</td>\n",
       "      <td>0.62</td>\n",
       "      <td>14122</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Malta</td>\n",
       "      <td>1.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Niue</td>\n",
       "      <td>Type F5</td>\n",
       "      <td>Quantum</td>\n",
       "      <td>ThetaBase Services</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>strict</td>\n",
       "      <td>1.0</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>467</td>\n",
       "      <td>4.66</td>\n",
       "      <td>47761</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Niue</td>\n",
       "      <td>2.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Malta</td>\n",
       "      <td>Type V2</td>\n",
       "      <td>Quantum</td>\n",
       "      <td>ThetaBase Services</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>moderate</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>318</td>\n",
       "      <td>3.22</td>\n",
       "      <td>26648</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Niue</td>\n",
       "      <td>2.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Malta</td>\n",
       "      <td>Type V2</td>\n",
       "      <td>Plasma</td>\n",
       "      <td>ThetaBase Services</td>\n",
       "      <td>5.0</td>\n",
       "      <td>10</td>\n",
       "      <td>strict</td>\n",
       "      <td>5.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>22</td>\n",
       "      <td>0.29</td>\n",
       "      <td>26648</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Niue</td>\n",
       "      <td>2.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "\n",
       "        shuttle_location shuttle_type engine_type       engine_vendor  \\\n",
       "\u001B[1;36m0\u001B[0m  Sao Tome and Principe      Type V5      Plasma  ThetaBase Services   \n",
       "\u001B[1;36m1\u001B[0m      Wallis and Futuna      Type V2      Plasma  ThetaBase Services   \n",
       "\u001B[1;36m2\u001B[0m                   Niue      Type F5     Quantum  ThetaBase Services   \n",
       "\u001B[1;36m3\u001B[0m                  Malta      Type V2     Quantum  ThetaBase Services   \n",
       "\u001B[1;36m4\u001B[0m                  Malta      Type V2      Plasma  ThetaBase Services   \n",
       "\n",
       "   engines  passenger_capacity cancellation_policy  crew  d_check_complete  \\\n",
       "\u001B[1;36m0\u001B[0m      \u001B[1;36m2.0\u001B[0m                   \u001B[1;36m4\u001B[0m            moderate   \u001B[1;36m2.0\u001B[0m             \u001B[3;91mFalse\u001B[0m   \n",
       "\u001B[1;36m1\u001B[0m      \u001B[1;36m3.0\u001B[0m                   \u001B[1;36m5\u001B[0m            moderate   \u001B[1;36m3.0\u001B[0m             \u001B[3;91mFalse\u001B[0m   \n",
       "\u001B[1;36m2\u001B[0m      \u001B[1;36m1.0\u001B[0m                   \u001B[1;36m2\u001B[0m              strict   \u001B[1;36m1.0\u001B[0m              \u001B[3;92mTrue\u001B[0m   \n",
       "\u001B[1;36m3\u001B[0m      \u001B[1;36m1.0\u001B[0m                   \u001B[1;36m2\u001B[0m            moderate   \u001B[1;36m1.0\u001B[0m             \u001B[3;91mFalse\u001B[0m   \n",
       "\u001B[1;36m4\u001B[0m      \u001B[1;36m5.0\u001B[0m                  \u001B[1;36m10\u001B[0m              strict   \u001B[1;36m5.0\u001B[0m             \u001B[3;91mFalse\u001B[0m   \n",
       "\n",
       "   moon_clearance_complete  \u001B[33m...\u001B[0m  review_scores_crew  review_scores_location  \\\n",
       "\u001B[1;36m0\u001B[0m                    \u001B[3;91mFalse\u001B[0m  \u001B[33m...\u001B[0m                 \u001B[1;36m9.0\u001B[0m                     \u001B[1;36m9.0\u001B[0m   \n",
       "\u001B[1;36m1\u001B[0m                    \u001B[3;91mFalse\u001B[0m  \u001B[33m...\u001B[0m                \u001B[1;36m10.0\u001B[0m                    \u001B[1;36m10.0\u001B[0m   \n",
       "\u001B[1;36m2\u001B[0m                    \u001B[3;91mFalse\u001B[0m  \u001B[33m...\u001B[0m                \u001B[1;36m10.0\u001B[0m                    \u001B[1;36m10.0\u001B[0m   \n",
       "\u001B[1;36m3\u001B[0m                    \u001B[3;91mFalse\u001B[0m  \u001B[33m...\u001B[0m                \u001B[1;36m10.0\u001B[0m                     \u001B[1;36m9.0\u001B[0m   \n",
       "\u001B[1;36m4\u001B[0m                    \u001B[3;91mFalse\u001B[0m  \u001B[33m...\u001B[0m                \u001B[1;36m10.0\u001B[0m                     \u001B[1;36m9.0\u001B[0m   \n",
       "\n",
       "   review_scores_price  number_of_reviews  reviews_per_month     id  \\\n",
       "\u001B[1;36m0\u001B[0m                  \u001B[1;36m9.0\u001B[0m                 \u001B[1;36m26\u001B[0m               \u001B[1;36m0.77\u001B[0m  \u001B[1;36m32413\u001B[0m   \n",
       "\u001B[1;36m1\u001B[0m                  \u001B[1;36m9.0\u001B[0m                 \u001B[1;36m61\u001B[0m               \u001B[1;36m0.62\u001B[0m  \u001B[1;36m14122\u001B[0m   \n",
       "\u001B[1;36m2\u001B[0m                 \u001B[1;36m10.0\u001B[0m                \u001B[1;36m467\u001B[0m               \u001B[1;36m4.66\u001B[0m  \u001B[1;36m47761\u001B[0m   \n",
       "\u001B[1;36m3\u001B[0m                  \u001B[1;36m9.0\u001B[0m                \u001B[1;36m318\u001B[0m               \u001B[1;36m3.22\u001B[0m  \u001B[1;36m26648\u001B[0m   \n",
       "\u001B[1;36m4\u001B[0m                 \u001B[1;36m10.0\u001B[0m                 \u001B[1;36m22\u001B[0m               \u001B[1;36m0.29\u001B[0m  \u001B[1;36m26648\u001B[0m   \n",
       "\n",
       "   company_rating  company_location  total_fleet_count  iata_approved  \n",
       "\u001B[1;36m0\u001B[0m             \u001B[1;36m1.0\u001B[0m     Faroe Islands                \u001B[1;36m1.0\u001B[0m          \u001B[3;91mFalse\u001B[0m  \n",
       "\u001B[1;36m1\u001B[0m             \u001B[1;36m1.0\u001B[0m             Malta                \u001B[1;36m1.0\u001B[0m           \u001B[3;92mTrue\u001B[0m  \n",
       "\u001B[1;36m2\u001B[0m             \u001B[1;36m1.0\u001B[0m              Niue                \u001B[1;36m2.0\u001B[0m          \u001B[3;91mFalse\u001B[0m  \n",
       "\u001B[1;36m3\u001B[0m             \u001B[1;36m1.0\u001B[0m              Niue                \u001B[1;36m2.0\u001B[0m           \u001B[3;92mTrue\u001B[0m  \n",
       "\u001B[1;36m4\u001B[0m             \u001B[1;36m1.0\u001B[0m              Niue                \u001B[1;36m2.0\u001B[0m           \u001B[3;92mTrue\u001B[0m  \n",
       "\n",
       "\u001B[1m[\u001B[0m\u001B[1;36m5\u001B[0m rows x \u001B[1;36m27\u001B[0m columns\u001B[1m]\u001B[0m"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from kedro.framework.session import KedroSession\n",
    "from kedro.framework.startup import bootstrap_project\n",
    "\n",
    "project_path = \"../kedro-code\"\n",
    "bootstrap_project(project_path)\n",
    "with KedroSession.create(project_path) as session:\n",
    "    context = session.load_context()\n",
    "    catalog = context.catalog\n",
    "    params = context.params\n",
    "\n",
    "inputs = dict(\n",
    "    companies=catalog.load(\"companies\"),\n",
    "    reviews=catalog.load(\"reviews\"),\n",
    "    shuttles=catalog.load(\"shuttles\"),\n",
    "    **params,  # unpack a dictionary\n",
    ")\n",
    "\n",
    "results = dr.execute([\"model_input_table\"], inputs=inputs)\n",
    "results[\"model_input_table\"].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Use Hamilton materializers\n",
    "\n",
    "The concept of [materialization](https://hamilton.apache.org/concepts/materialization/#) in Hamilton closely ressembles `Dataset` objects in Kedro. The `from_` object defines loaders and `to` defines savers, and they're collectively called \"materializers\".\n",
    "\n",
    "If you're using Hamilton, you'll find plenty of savers & loaders for common formats and libraries (`parquet`, `csv`, `json`, `xgboost`, `dlt`, etc.). We suggest using them directly if you're wanting to move away from the data catalog and YAML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\n",
       "\u001B[1m{\u001B[0m\n",
       "    \u001B[32m'model_input_table__parquet'\u001B[0m: \u001B[1m{\u001B[0m\n",
       "        \u001B[32m'file_metadata'\u001B[0m: \u001B[1m{\u001B[0m\n",
       "            \u001B[32m'size'\u001B[0m: \u001B[1;36m215150\u001B[0m,\n",
       "            \u001B[32m'path'\u001B[0m: \u001B[32m'../kedro-code/data/03_primary/model_input_table.pq'\u001B[0m,\n",
       "            \u001B[32m'last_modified'\u001B[0m: \u001B[1;36m1716408659.312288\u001B[0m,\n",
       "            \u001B[32m'timestamp'\u001B[0m: \u001B[1;36m1716423059.328785\u001B[0m,\n",
       "            \u001B[32m'scheme'\u001B[0m: \u001B[32m''\u001B[0m,\n",
       "            \u001B[32m'notes'\u001B[0m: \u001B[32m''\u001B[0m\n",
       "        \u001B[1m}\u001B[0m,\n",
       "        \u001B[32m'dataframe_metadata'\u001B[0m: \u001B[1m{\u001B[0m\n",
       "            \u001B[32m'rows'\u001B[0m: \u001B[1;36m6027\u001B[0m,\n",
       "            \u001B[32m'columns'\u001B[0m: \u001B[1;36m27\u001B[0m,\n",
       "            \u001B[32m'column_names'\u001B[0m: \u001B[1m[\u001B[0m\n",
       "                \u001B[32m'shuttle_location'\u001B[0m,\n",
       "                \u001B[32m'shuttle_type'\u001B[0m,\n",
       "                \u001B[32m'engine_type'\u001B[0m,\n",
       "                \u001B[32m'engine_vendor'\u001B[0m,\n",
       "                \u001B[32m'engines'\u001B[0m,\n",
       "                \u001B[32m'passenger_capacity'\u001B[0m,\n",
       "                \u001B[32m'cancellation_policy'\u001B[0m,\n",
       "                \u001B[32m'crew'\u001B[0m,\n",
       "                \u001B[32m'd_check_complete'\u001B[0m,\n",
       "                \u001B[32m'moon_clearance_complete'\u001B[0m,\n",
       "                \u001B[32m'price'\u001B[0m,\n",
       "                \u001B[32m'company_id'\u001B[0m,\n",
       "                \u001B[32m'shuttle_id'\u001B[0m,\n",
       "                \u001B[32m'review_scores_rating'\u001B[0m,\n",
       "                \u001B[32m'review_scores_comfort'\u001B[0m,\n",
       "                \u001B[32m'review_scores_amenities'\u001B[0m,\n",
       "                \u001B[32m'review_scores_trip'\u001B[0m,\n",
       "                \u001B[32m'review_scores_crew'\u001B[0m,\n",
       "                \u001B[32m'review_scores_location'\u001B[0m,\n",
       "                \u001B[32m'review_scores_price'\u001B[0m,\n",
       "                \u001B[32m'number_of_reviews'\u001B[0m,\n",
       "                \u001B[32m'reviews_per_month'\u001B[0m,\n",
       "                \u001B[32m'id'\u001B[0m,\n",
       "                \u001B[32m'company_rating'\u001B[0m,\n",
       "                \u001B[32m'company_location'\u001B[0m,\n",
       "                \u001B[32m'total_fleet_count'\u001B[0m,\n",
       "                \u001B[32m'iata_approved'\u001B[0m\n",
       "            \u001B[1m]\u001B[0m,\n",
       "            \u001B[32m'datatypes'\u001B[0m: \u001B[1m[\u001B[0m\n",
       "                \u001B[32m'string'\u001B[0m,\n",
       "                \u001B[32m'string'\u001B[0m,\n",
       "                \u001B[32m'string'\u001B[0m,\n",
       "                \u001B[32m'string'\u001B[0m,\n",
       "                \u001B[32m'Int64'\u001B[0m,\n",
       "                \u001B[32m'Int64'\u001B[0m,\n",
       "                \u001B[32m'string'\u001B[0m,\n",
       "                \u001B[32m'Int64'\u001B[0m,\n",
       "                \u001B[32m'boolean'\u001B[0m,\n",
       "                \u001B[32m'boolean'\u001B[0m,\n",
       "                \u001B[32m'float64'\u001B[0m,\n",
       "                \u001B[32m'Int64'\u001B[0m,\n",
       "                \u001B[32m'Int64'\u001B[0m,\n",
       "                \u001B[32m'Float64'\u001B[0m,\n",
       "                \u001B[32m'Float64'\u001B[0m,\n",
       "                \u001B[32m'Float64'\u001B[0m,\n",
       "                \u001B[32m'Float64'\u001B[0m,\n",
       "                \u001B[32m'Float64'\u001B[0m,\n",
       "                \u001B[32m'Float64'\u001B[0m,\n",
       "                \u001B[32m'Float64'\u001B[0m,\n",
       "                \u001B[32m'Int64'\u001B[0m,\n",
       "                \u001B[32m'Float64'\u001B[0m,\n",
       "                \u001B[32m'Int64'\u001B[0m,\n",
       "                \u001B[32m'float64'\u001B[0m,\n",
       "                \u001B[32m'string'\u001B[0m,\n",
       "                \u001B[32m'Float64'\u001B[0m,\n",
       "                \u001B[32m'boolean'\u001B[0m\n",
       "            \u001B[1m]\u001B[0m\n",
       "        \u001B[1m}\u001B[0m\n",
       "    \u001B[1m}\u001B[0m\n",
       "\u001B[1m}\u001B[0m"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from hamilton.io.materialization import from_, to\n",
    "\n",
    "# contains both Savers and Loaders\n",
    "materializers = [\n",
    "    # `target` is the name of the Hamilton node receiving data\n",
    "    from_.csv(\n",
    "        target=\"companies\",\n",
    "        path=\"../kedro-code/data/01_raw/companies.csv\",\n",
    "    ), \n",
    "    from_.csv(\n",
    "        target=\"reviews\",\n",
    "        path=\"../kedro-code/data/01_raw/reviews.csv\",\n",
    "    ), \n",
    "    from_.excel(\n",
    "        target=\"shuttles\",\n",
    "        path=\"../kedro-code/data/01_raw/shuttles.xlsx\",\n",
    "    ),\n",
    "    # `id` is the name of the generated \"saver\" node\n",
    "    to.parquet(\n",
    "        id=\"model_input_table__parquet\",\n",
    "        dependencies=[\"model_input_table\"],\n",
    "        path=\"../kedro-code/data/03_primary/model_input_table.pq\",\n",
    "    )\n",
    "]\n",
    "\n",
    "# `.materialize()` will load data using `from_` objects and store results of `to` objects\n",
    "# it returns execution metadata about stored results.\n",
    "metadata, _ = dr.materialize(*materializers)\n",
    "metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If Hamilton doesn't support a specific format, you can use Kedro datasets using the `from_.kedro()` and `to.kedro()` materializers!\n",
    "\n",
    "This first snippet shows how to read from the `DataCatalog` defined in YAML format within the Kedro project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[05/22/24 16:10:59] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">shuttles</span> <span style=\"font-weight: bold\">(</span>ExcelDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                       <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m[05/22/24 16:10:59]\u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mshuttles\u001B[0m \u001B[1m(\u001B[0mExcelDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                       \u001B]8;id=251889;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=569579;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[05/22/24 16:11:01] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">companies</span> <span style=\"font-weight: bold\">(</span>CSVDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                        <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m[05/22/24 16:11:01]\u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mcompanies\u001B[0m \u001B[1m(\u001B[0mCSVDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                        \u001B]8;id=698982;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=247154;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">reviews</span> <span style=\"font-weight: bold\">(</span>CSVDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                          <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m                   \u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mreviews\u001B[0m \u001B[1m(\u001B[0mCSVDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                          \u001B]8;id=651853;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=754005;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[05/22/24 16:11:02] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Saving data to <span style=\"color: #ff8700; text-decoration-color: #ff8700\">model_input_table</span> <span style=\"font-weight: bold\">(</span>ParquetDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>               <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#525\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">525</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m[05/22/24 16:11:02]\u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Saving data to \u001B[38;5;208mmodel_input_table\u001B[0m \u001B[1m(\u001B[0mParquetDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m               \u001B]8;id=845498;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=59594;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#525\u001B\\\u001B[2m525\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\u001B[1m{\u001B[0m\u001B[32m'model_input_table__parquet'\u001B[0m: \u001B[1m{\u001B[0m\u001B[32m'success'\u001B[0m: \u001B[3;92mTrue\u001B[0m\u001B[1m}\u001B[0m\u001B[1m}\u001B[0m"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project_path = \"../kedro-code\"\n",
    "bootstrap_project(project_path)\n",
    "with KedroSession.create(project_path) as session:\n",
    "    context = session.load_context()\n",
    "    catalog = context.catalog\n",
    "    params = context.params\n",
    "\n",
    "# pass the `DataCatalog` to the `catalog` parameter\n",
    "materializers = [\n",
    "    from_.kedro(\n",
    "        target=\"companies\",\n",
    "        dataset_name=\"companies\",\n",
    "        catalog=catalog,\n",
    "    ),\n",
    "    from_.kedro(\n",
    "        target=\"shuttles\",\n",
    "        dataset_name=\"shuttles\",\n",
    "        catalog=catalog,\n",
    "    ),\n",
    "    from_.kedro(\n",
    "        target=\"reviews\",\n",
    "        dataset_name=\"reviews\",\n",
    "        catalog=catalog,\n",
    "    ),\n",
    "    to.kedro(\n",
    "        id=\"model_input_table__parquet\",\n",
    "        dependencies=[\"model_input_table\"],\n",
    "        dataset_name=\"model_input_table\",\n",
    "        catalog=catalog,\n",
    "    ),\n",
    "]\n",
    "metadata, _ = dr.materialize(*materializers)\n",
    "metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This second snippet shows how to define a `DataCatalog` directly in Python ([documentation](https://docs.kedro.org/en/stable/data/advanced_data_catalog_usage.html)). Note that in this case, all formats are already supported in Hamilton and it makes code a bit redundant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">shuttles</span> <span style=\"font-weight: bold\">(</span>ExcelDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                       <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m                   \u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mshuttles\u001B[0m \u001B[1m(\u001B[0mExcelDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                       \u001B]8;id=765638;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=540969;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[05/22/24 16:11:04] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">companies</span> <span style=\"font-weight: bold\">(</span>CSVDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                        <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m[05/22/24 16:11:04]\u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mcompanies\u001B[0m \u001B[1m(\u001B[0mCSVDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                        \u001B]8;id=845539;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=373818;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">reviews</span> <span style=\"font-weight: bold\">(</span>CSVDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                          <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m                   \u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mreviews\u001B[0m \u001B[1m(\u001B[0mCSVDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                          \u001B]8;id=673966;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=380648;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Saving data to <span style=\"color: #ff8700; text-decoration-color: #ff8700\">model_input_table</span> <span style=\"font-weight: bold\">(</span>ParquetDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>               <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#525\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">525</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m                   \u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Saving data to \u001B[38;5;208mmodel_input_table\u001B[0m \u001B[1m(\u001B[0mParquetDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m               \u001B]8;id=611834;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=912002;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#525\u001B\\\u001B[2m525\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\u001B[1m{\u001B[0m\u001B[32m'model_input_table__parquet'\u001B[0m: \u001B[1m{\u001B[0m\u001B[32m'success'\u001B[0m: \u001B[3;92mTrue\u001B[0m\u001B[1m}\u001B[0m\u001B[1m}\u001B[0m"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from kedro.io import DataCatalog\n",
    "from kedro_datasets.pandas import (\n",
    "    CSVDataset,\n",
    "    ExcelDataset,\n",
    "    ParquetDataset,\n",
    ")\n",
    "\n",
    "catalog = DataCatalog(dict(\n",
    "    companies=CSVDataset(filepath=\"../kedro-code/data/01_raw/companies.csv\"),\n",
    "    reviews=CSVDataset(filepath=\"../kedro-code/data/01_raw/reviews.csv\"),\n",
    "    shuttles=ExcelDataset(filepath=\"../kedro-code/data/01_raw/shuttles.xlsx\"),\n",
    "    model_input_table=ParquetDataset(filepath=\"../kedro-code/data/03_primary/model_input_table.pq\")\n",
    "))\n",
    "\n",
    "materializers = [\n",
    "    from_.kedro(\n",
    "        target=\"companies\",\n",
    "        dataset_name=\"companies\",\n",
    "        catalog=catalog,\n",
    "    ),\n",
    "    from_.kedro(\n",
    "        target=\"shuttles\",\n",
    "        dataset_name=\"shuttles\",\n",
    "        catalog=catalog,\n",
    "    ),\n",
    "    from_.kedro(\n",
    "        target=\"reviews\",\n",
    "        dataset_name=\"reviews\",\n",
    "        catalog=catalog,\n",
    "    ),\n",
    "    to.kedro(\n",
    "        id=\"model_input_table__parquet\",\n",
    "        dependencies=[\"model_input_table\"],\n",
    "        dataset_name=\"model_input_table\",\n",
    "        catalog=catalog,\n",
    "    ),\n",
    "]\n",
    "metadata, _ = dr.materialize(*materializers)\n",
    "metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Connect to the Hamilton UI\n",
    "\n",
    "To connect Hamilton to the Hamilton UI, we need to add the `HamiltonTracker` adapter. To connect your Kedro `Pipeline`, you need to pass a `Builder` containing the tracker when converting it to a Hamilton `Driver`. \n",
    "\n",
    "The next cell will install the dependencies for the UI. You also need the Hamilton UI container running for the next cells to successfully execute and send metadata. See [installation instructions](https://hamilton.apache.org/concepts/ui/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install hamilton_sdk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the next cell and creating the `Driver` should populate the Hamilton UI with the dataflow definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"1220pt\" height=\"372pt\"\n",
       " viewBox=\"0.00 0.00 1220.00 372.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 368)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-368 1216,-368 1216,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"46,-224 46,-356 142,-356 142,-224 46,-224\"/>\n",
       "<text text-anchor=\"middle\" x=\"94\" y=\"-340.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- preprocessed_shuttles -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>preprocessed_shuttles</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M415,-78C415,-78 239,-78 239,-78 233,-78 227,-72 227,-66 227,-66 227,-26 227,-26 227,-20 233,-14 239,-14 239,-14 415,-14 415,-14 421,-14 427,-20 427,-26 427,-26 427,-66 427,-66 427,-72 421,-78 415,-78\"/>\n",
       "<text text-anchor=\"start\" x=\"238\" y=\"-56.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">preprocessed_shuttles</text>\n",
       "<text text-anchor=\"start\" x=\"288.5\" y=\"-28.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- model_input_table -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>model_input_table</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M619,-151C619,-151 478,-151 478,-151 472,-151 466,-145 466,-139 466,-139 466,-99 466,-99 466,-93 472,-87 478,-87 478,-87 619,-87 619,-87 625,-87 631,-93 631,-99 631,-99 631,-139 631,-139 631,-145 625,-151 619,-151\"/>\n",
       "<text text-anchor=\"start\" x=\"477\" y=\"-129.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">model_input_table</text>\n",
       "<text text-anchor=\"start\" x=\"510\" y=\"-101.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- preprocessed_shuttles&#45;&gt;model_input_table -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>preprocessed_shuttles&#45;&gt;model_input_table</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M424.28,-78.02C434.86,-81.54 445.61,-85.11 456.13,-88.61\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"455.12,-91.97 465.72,-91.8 457.33,-85.32 455.12,-91.97\"/>\n",
       "</g>\n",
       "<!-- X_test -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>X_test</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M850,-310C850,-310 803,-310 803,-310 797,-310 791,-304 791,-298 791,-298 791,-258 791,-258 791,-252 797,-246 803,-246 803,-246 850,-246 850,-246 856,-246 862,-252 862,-258 862,-258 862,-298 862,-298 862,-304 856,-310 850,-310\"/>\n",
       "<text text-anchor=\"start\" x=\"802\" y=\"-288.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">X_test</text>\n",
       "<text text-anchor=\"start\" x=\"812.5\" y=\"-260.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Any</text>\n",
       "</g>\n",
       "<!-- evaluate_model -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>evaluate_model</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M1200,-172C1200,-172 1078,-172 1078,-172 1072,-172 1066,-166 1066,-160 1066,-160 1066,-120 1066,-120 1066,-114 1072,-108 1078,-108 1078,-108 1200,-108 1200,-108 1206,-108 1212,-114 1212,-120 1212,-120 1212,-160 1212,-160 1212,-166 1206,-172 1200,-172\"/>\n",
       "<text text-anchor=\"start\" x=\"1077\" y=\"-150.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">evaluate_model</text>\n",
       "<text text-anchor=\"start\" x=\"1104\" y=\"-122.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">NoneType</text>\n",
       "</g>\n",
       "<!-- X_test&#45;&gt;evaluate_model -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>X_test&#45;&gt;evaluate_model</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M862.21,-262.57C908.97,-241.79 993.61,-204.17 1056.83,-176.07\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1058.28,-179.26 1066,-172 1055.44,-172.86 1058.28,-179.26\"/>\n",
       "</g>\n",
       "<!-- X_train -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>X_train</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M853,-228C853,-228 800,-228 800,-228 794,-228 788,-222 788,-216 788,-216 788,-176 788,-176 788,-170 794,-164 800,-164 800,-164 853,-164 853,-164 859,-164 865,-170 865,-176 865,-176 865,-216 865,-216 865,-222 859,-228 853,-228\"/>\n",
       "<text text-anchor=\"start\" x=\"799\" y=\"-206.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">X_train</text>\n",
       "<text text-anchor=\"start\" x=\"812.5\" y=\"-178.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Any</text>\n",
       "</g>\n",
       "<!-- regressor -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>regressor</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M1025,-172C1025,-172 906,-172 906,-172 900,-172 894,-166 894,-160 894,-160 894,-120 894,-120 894,-114 900,-108 906,-108 906,-108 1025,-108 1025,-108 1031,-108 1037,-114 1037,-120 1037,-120 1037,-160 1037,-160 1037,-166 1031,-172 1025,-172\"/>\n",
       "<text text-anchor=\"start\" x=\"927.5\" y=\"-150.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">regressor</text>\n",
       "<text text-anchor=\"start\" x=\"905\" y=\"-122.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">LinearRegression</text>\n",
       "</g>\n",
       "<!-- X_train&#45;&gt;regressor -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>X_train&#45;&gt;regressor</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M865.3,-180.55C871.31,-178.09 877.72,-175.47 884.28,-172.79\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"886.04,-175.85 893.97,-168.83 883.39,-169.37 886.04,-175.85\"/>\n",
       "</g>\n",
       "<!-- regressor&#45;&gt;evaluate_model -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>regressor&#45;&gt;evaluate_model</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1037.25,-140C1043.27,-140 1049.41,-140 1055.52,-140\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1055.75,-143.5 1065.75,-140 1055.75,-136.5 1055.75,-143.5\"/>\n",
       "</g>\n",
       "<!-- y_test -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>y_test</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M849.5,-64C849.5,-64 803.5,-64 803.5,-64 797.5,-64 791.5,-58 791.5,-52 791.5,-52 791.5,-12 791.5,-12 791.5,-6 797.5,0 803.5,0 803.5,0 849.5,0 849.5,0 855.5,0 861.5,-6 861.5,-12 861.5,-12 861.5,-52 861.5,-52 861.5,-58 855.5,-64 849.5,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"802.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">y_test</text>\n",
       "<text text-anchor=\"start\" x=\"812.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Any</text>\n",
       "</g>\n",
       "<!-- y_test&#45;&gt;evaluate_model -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>y_test&#45;&gt;evaluate_model</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M861.51,-42.01C903.18,-54.47 975.79,-76.82 1037,-99 1043.34,-101.3 1049.86,-103.75 1056.41,-106.26\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1055.4,-109.62 1065.98,-109.99 1057.93,-103.1 1055.4,-109.62\"/>\n",
       "</g>\n",
       "<!-- split_data -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>split_data</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M747,-187C747,-187 672,-187 672,-187 666,-187 660,-181 660,-175 660,-175 660,-135 660,-135 660,-129 666,-123 672,-123 672,-123 747,-123 747,-123 753,-123 759,-129 759,-135 759,-135 759,-175 759,-175 759,-181 753,-187 747,-187\"/>\n",
       "<text text-anchor=\"start\" x=\"671\" y=\"-165.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">split_data</text>\n",
       "<text text-anchor=\"start\" x=\"695.5\" y=\"-137.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Dict</text>\n",
       "</g>\n",
       "<!-- model_input_table&#45;&gt;split_data -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>model_input_table&#45;&gt;split_data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M631.33,-137.53C637.57,-138.94 643.79,-140.35 649.84,-141.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"649.3,-145.18 659.82,-143.98 650.84,-138.36 649.3,-145.18\"/>\n",
       "</g>\n",
       "<!-- preprocessed_companies -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>preprocessed_companies</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M425,-224C425,-224 229,-224 229,-224 223,-224 217,-218 217,-212 217,-212 217,-172 217,-172 217,-166 223,-160 229,-160 229,-160 425,-160 425,-160 431,-160 437,-166 437,-172 437,-172 437,-212 437,-212 437,-218 431,-224 425,-224\"/>\n",
       "<text text-anchor=\"start\" x=\"228\" y=\"-202.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">preprocessed_companies</text>\n",
       "<text text-anchor=\"start\" x=\"288.5\" y=\"-174.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- preprocessed_companies&#45;&gt;model_input_table -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>preprocessed_companies&#45;&gt;model_input_table</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M424.28,-159.98C434.86,-156.46 445.61,-152.89 456.13,-149.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"457.33,-152.68 465.72,-146.2 455.12,-146.03 457.33,-152.68\"/>\n",
       "</g>\n",
       "<!-- y_train -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>y_train</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M852.5,-146C852.5,-146 800.5,-146 800.5,-146 794.5,-146 788.5,-140 788.5,-134 788.5,-134 788.5,-94 788.5,-94 788.5,-88 794.5,-82 800.5,-82 800.5,-82 852.5,-82 852.5,-82 858.5,-82 864.5,-88 864.5,-94 864.5,-94 864.5,-134 864.5,-134 864.5,-140 858.5,-146 852.5,-146\"/>\n",
       "<text text-anchor=\"start\" x=\"799.5\" y=\"-124.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">y_train</text>\n",
       "<text text-anchor=\"start\" x=\"812.5\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Any</text>\n",
       "</g>\n",
       "<!-- y_train&#45;&gt;regressor -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>y_train&#45;&gt;regressor</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M864.57,-121.04C870.7,-122.2 877.25,-123.44 883.96,-124.71\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"883.39,-128.17 893.87,-126.6 884.7,-121.29 883.39,-128.17\"/>\n",
       "</g>\n",
       "<!-- split_data&#45;&gt;X_test -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>split_data&#45;&gt;X_test</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M740.71,-187.32C755.37,-202.99 773.11,-221.97 788.57,-238.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"786.1,-240.98 795.49,-245.9 791.21,-236.2 786.1,-240.98\"/>\n",
       "</g>\n",
       "<!-- split_data&#45;&gt;X_train -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>split_data&#45;&gt;X_train</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M759.27,-172.39C765.54,-174.62 771.95,-176.91 778.19,-179.13\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"777.23,-182.5 787.82,-182.57 779.58,-175.91 777.23,-182.5\"/>\n",
       "</g>\n",
       "<!-- split_data&#45;&gt;y_test -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>split_data&#45;&gt;y_test</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M740.71,-122.68C755.37,-107.01 773.11,-88.03 788.57,-71.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"791.21,-73.8 795.49,-64.1 786.1,-69.02 791.21,-73.8\"/>\n",
       "</g>\n",
       "<!-- split_data&#45;&gt;y_train -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>split_data&#45;&gt;y_train</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M759.27,-137.61C765.62,-135.35 772.11,-133.03 778.43,-130.78\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"779.93,-133.96 788.17,-127.31 777.58,-127.37 779.93,-133.96\"/>\n",
       "</g>\n",
       "<!-- _preprocessed_shuttles_inputs -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>_preprocessed_shuttles_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"178,-68.5 10,-68.5 10,-23.5 178,-23.5 178,-68.5\"/>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-41.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">shuttles</text>\n",
       "<text text-anchor=\"start\" x=\"87\" y=\"-41.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- _preprocessed_shuttles_inputs&#45;&gt;preprocessed_shuttles -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_preprocessed_shuttles_inputs&#45;&gt;preprocessed_shuttles</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M178.07,-46C190.52,-46 203.52,-46 216.42,-46\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"216.73,-49.5 226.73,-46 216.73,-42.5 216.73,-49.5\"/>\n",
       "</g>\n",
       "<!-- _model_input_table_inputs -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>_model_input_table_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"409.5,-141.5 244.5,-141.5 244.5,-96.5 409.5,-96.5 409.5,-141.5\"/>\n",
       "<text text-anchor=\"start\" x=\"260\" y=\"-114.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">reviews</text>\n",
       "<text text-anchor=\"start\" x=\"319\" y=\"-114.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- _model_input_table_inputs&#45;&gt;model_input_table -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>_model_input_table_inputs&#45;&gt;model_input_table</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M409.73,-119C424.59,-119 440.18,-119 455.31,-119\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"455.7,-122.5 465.7,-119 455.7,-115.5 455.7,-122.5\"/>\n",
       "</g>\n",
       "<!-- _preprocessed_companies_inputs -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>_preprocessed_companies_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"188,-214.5 0,-214.5 0,-169.5 188,-169.5 188,-214.5\"/>\n",
       "<text text-anchor=\"start\" x=\"15\" y=\"-187.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">companies</text>\n",
       "<text text-anchor=\"start\" x=\"97\" y=\"-187.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- _preprocessed_companies_inputs&#45;&gt;preprocessed_companies -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>_preprocessed_companies_inputs&#45;&gt;preprocessed_companies</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M188.29,-192C194.31,-192 200.41,-192 206.52,-192\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"206.81,-195.5 216.81,-192 206.81,-188.5 206.81,-195.5\"/>\n",
       "</g>\n",
       "<!-- _split_data_inputs -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>_split_data_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"631,-214.5 466,-214.5 466,-169.5 631,-169.5 631,-214.5\"/>\n",
       "<text text-anchor=\"start\" x=\"481.5\" y=\"-187.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">model_options</text>\n",
       "<text text-anchor=\"start\" x=\"588.5\" y=\"-187.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Dict</text>\n",
       "</g>\n",
       "<!-- _split_data_inputs&#45;&gt;split_data -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>_split_data_inputs&#45;&gt;split_data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M631.33,-172.96C637.57,-171.51 643.79,-170.06 649.84,-168.65\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"650.88,-172 659.82,-166.33 649.29,-165.19 650.88,-172\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"123.5,-324.5 64.5,-324.5 64.5,-287.5 123.5,-287.5 123.5,-324.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"94\" y=\"-302.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M122,-269.5C122,-269.5 66,-269.5 66,-269.5 60,-269.5 54,-263.5 54,-257.5 54,-257.5 54,-244.5 54,-244.5 54,-238.5 60,-232.5 66,-232.5 66,-232.5 122,-232.5 122,-232.5 128,-232.5 134,-238.5 134,-244.5 134,-244.5 134,-257.5 134,-257.5 134,-263.5 128,-269.5 122,-269.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"94\" y=\"-247.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "\u001B[1m<\u001B[0m\u001B[1;95mhamilton.driver.Driver\u001B[0m\u001B[39m object at \u001B[0m\u001B[1;36m0x7f267841b450\u001B[0m\u001B[1m>\u001B[0m"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from hamilton_sdk.adapters import HamiltonTracker\n",
    "from hamilton import driver\n",
    "from hamilton.plugins import h_kedro\n",
    "from kedro_code.pipelines import data_processing, data_science\n",
    "\n",
    "# modify this as needed\n",
    "tracker = HamiltonTracker(\n",
    "    project_id=3,\n",
    "    username=\"abc@my_domain.com\",\n",
    "    dag_name=\"spaceflight\",\n",
    ")\n",
    "builder = driver.Builder().with_adapters(tracker)\n",
    "\n",
    "dr = h_kedro.kedro_pipeline_to_driver(\n",
    "    data_processing.create_pipeline(),\n",
    "    data_science.create_pipeline(),\n",
    "    builder=builder\n",
    ")\n",
    "dr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling `.materialize()` in the next cell (or using `.execute()`) will populate the Hamilton UI with execution metadata and artifact introspection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[05/22/24 16:11:10] </span><span style=\"color: #800000; text-decoration-color: #800000\">WARNING </span>                                                                        <a href=\"file:///home/tjean/projects/dagworks/hamilton/ui/sdk/src/hamilton_sdk/adapters.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">adapters.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/ui/sdk/src/hamilton_sdk/adapters.py#163\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">163</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         Capturing execution run. Results can be found at                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0000ff; text-decoration-color: #0000ff; text-decoration: underline\">http://localhost:8242/dashboard/project/3/runs/32</span>                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                                                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m[05/22/24 16:11:10]\u001B[0m\u001B[2;36m \u001B[0m\u001B[31mWARNING \u001B[0m                                                                        \u001B]8;id=299905;file:///home/tjean/projects/dagworks/hamilton/ui/sdk/src/hamilton_sdk/adapters.py\u001B\\\u001B[2madapters.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=155714;file:///home/tjean/projects/dagworks/hamilton/ui/sdk/src/hamilton_sdk/adapters.py#163\u001B\\\u001B[2m163\u001B[0m\u001B]8;;\u001B\\\n",
       "\u001B[2;36m                    \u001B[0m         Capturing execution run. Results can be found at                       \u001B[2m               \u001B[0m\n",
       "\u001B[2;36m                    \u001B[0m         \u001B[4;94mhttp://localhost:8242/dashboard/project/3/runs/32\u001B[0m                      \u001B[2m               \u001B[0m\n",
       "\u001B[2;36m                    \u001B[0m                                                                                \u001B[2m               \u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">shuttles</span> <span style=\"font-weight: bold\">(</span>ExcelDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                       <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m                   \u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mshuttles\u001B[0m \u001B[1m(\u001B[0mExcelDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                       \u001B]8;id=976064;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=997503;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[05/22/24 16:11:13] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">companies</span> <span style=\"font-weight: bold\">(</span>CSVDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                        <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m[05/22/24 16:11:13]\u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mcompanies\u001B[0m \u001B[1m(\u001B[0mCSVDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                        \u001B]8;id=134319;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=446665;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loading data from <span style=\"color: #ff8700; text-decoration-color: #ff8700\">reviews</span> <span style=\"font-weight: bold\">(</span>CSVDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>                          <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">483</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m                   \u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Loading data from \u001B[38;5;208mreviews\u001B[0m \u001B[1m(\u001B[0mCSVDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m                          \u001B]8;id=620527;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=424729;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#483\u001B\\\u001B[2m483\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Saving data to <span style=\"color: #ff8700; text-decoration-color: #ff8700\">model_input_table</span> <span style=\"font-weight: bold\">(</span>ParquetDataset<span style=\"font-weight: bold\">)</span><span style=\"color: #808000; text-decoration-color: #808000\">...</span>               <a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">data_catalog.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#525\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">525</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m                   \u001B[0m\u001B[2;36m \u001B[0m\u001B[34mINFO    \u001B[0m Saving data to \u001B[38;5;208mmodel_input_table\u001B[0m \u001B[1m(\u001B[0mParquetDataset\u001B[1m)\u001B[0m\u001B[33m...\u001B[0m               \u001B]8;id=102118;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py\u001B\\\u001B[2mdata_catalog.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=182015;file:///home/tjean/projects/dagworks/hamilton/examples/kedro/venv/lib/python3.11/site-packages/kedro/io/data_catalog.py#525\u001B\\\u001B[2m525\u001B[0m\u001B]8;;\u001B\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #800000; text-decoration-color: #800000\">WARNING </span>                                                                        <a href=\"file:///home/tjean/projects/dagworks/hamilton/ui/sdk/src/hamilton_sdk/adapters.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">adapters.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///home/tjean/projects/dagworks/hamilton/ui/sdk/src/hamilton_sdk/adapters.py#352\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">352</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         Captured execution run. Results can be found at                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0000ff; text-decoration-color: #0000ff; text-decoration: underline\">http://localhost:8242/dashboard/project/3/runs/32</span>                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                                                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2;36m                   \u001B[0m\u001B[2;36m \u001B[0m\u001B[31mWARNING \u001B[0m                                                                        \u001B]8;id=48567;file:///home/tjean/projects/dagworks/hamilton/ui/sdk/src/hamilton_sdk/adapters.py\u001B\\\u001B[2madapters.py\u001B[0m\u001B]8;;\u001B\\\u001B[2m:\u001B[0m\u001B]8;id=639017;file:///home/tjean/projects/dagworks/hamilton/ui/sdk/src/hamilton_sdk/adapters.py#352\u001B\\\u001B[2m352\u001B[0m\u001B]8;;\u001B\\\n",
       "\u001B[2;36m                    \u001B[0m         Captured execution run. Results can be found at                        \u001B[2m               \u001B[0m\n",
       "\u001B[2;36m                    \u001B[0m         \u001B[4;94mhttp://localhost:8242/dashboard/project/3/runs/32\u001B[0m                      \u001B[2m               \u001B[0m\n",
       "\u001B[2;36m                    \u001B[0m                                                                                \u001B[2m               \u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# reuse the previously defined materializers\n",
    "_, _ = dr.materialize(*materializers)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
